Skip to content

fix(ci): wait for controller reconciliation before Kubernetes conformance - #4676

Merged
TaylorMutch merged 1 commit into
mainfrom
4675-kubernetes-conformance-startup/pimlock
Oct 20, 2026
Merged

TaylorMutch merged 1 commit into
mainfrom
4675-kubernetes-conformance-startup/pimlock

Conversation

@pimlock

@pimlock pimlock commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Kubernetes conformance currently starts when the gateway connects, even if Agent Sandbox is still recovering after guest boot. This can fail the first sandbox creation before scenario assertions run. This PR waits for controller reconciliation before starting conformance and collects controller diagnostics when preparation or tests fail.

Related Issue

Closes #4675.

Changes

  • Verify reconciliation with a temporary Sandbox and its controller-owned Pod. A scheduling gate prevents the Pod from running or pulling an image.
  • Retry temporary API failures and remove the probe and dependent resources on success or failure. Skip conformance and collect diagnostics if preparation fails.
  • Collect controller state, leader leases, events, and bounded current and previous logs. Retain startup and recent log sections.
  • Update the Actions troubleshooting skill with controller diagnostics guidance. Production provisioning deadlines and ownership checks remain unchanged.

Testing

  • All eight archived conformance tests passed in 298.4 seconds on an ARM64 Ubuntu 24.04 guest with K3s 1.29.15 and Agent Sandbox 0.5.0. Runtime artifacts came from be9d7a0561cfd55b1ca426f0217e283a4e28b6b8. The probe waited 10.9 seconds through controller recovery, then cleaned up.
  • Isolated checks covered delayed leadership, stopped-controller cleanup and conformance skipping, and retry after a temporary apply failure.
  • The diagnostic collector ran in the VM. A 400 KB log fixture retained startup/recent markers and command failure status within the output limit. ShellCheck passed.

Local VM validation used ARM64; the original CI failure used x86-64. The local controller certificate error and recovery do not establish the original failure's exact cause.

Checklist

  • Follows Conventional Commits.
  • Commits are signed off for DCO.
  • Contributor troubleshooting guidance updated; runtime architecture is unchanged.

@TaylorMutch

Copy link
Copy Markdown
Collaborator

@pimlock can you resolve the conflict? Otherwise LGTM

@TaylorMutch TaylorMutch self-assigned this Oct 19, 2026
…ance

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
pimlock force-pushed the 4675-kubernetes-conformance-startup/pimlock branch from d37b595 to a816e30 Compare October 19, 2026 23:28
@pimlock pimlock added the test:e2e Requires end-to-end coverage label Oct 19, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for a816e30. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the full diff; no blocking findings. The scheduling-gated probe verifies controller ownership and reconciliation, cleanup runs on success and failure, and readiness failures collect diagnostics before failing conformance. ShellCheck and Bash syntax checks passed for the diagnostic collector. All PR checks completed: 109 passed and 12 skipped, including passing K3s and K3s HA installed-artifact conformance.

@TaylorMutch
TaylorMutch added this pull request to the merge queue Oct 20, 2026
Merged via the queue into main with commit 7417f75 Oct 20, 2026
155 checks passed
@TaylorMutch
TaylorMutch deleted the 4675-kubernetes-conformance-startup/pimlock branch October 20, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(ci): Kubernetes conformance starts before controller reconciliation recovers

2 participants